Document clustering based on keyword frequency and concept matching technique in Hadoop

. The term big data has come into use in recent years. It is used to refer to the ever-increasing amount of information that organisations are storing, processing and analysing. Owing to the growing number of information sources big data based file systems are necessary. Due to the explosion growth of digital information, automatic document clustering or categorization has become more important. Document management and clustering is more important for content management systems. It is a necessary to mine and store web documents in CMS. In this study online document based content management with automatic URL indexing is dealt with. It’s highly possible that the content in CMS will be redundant over the web as most of the time the content will be gathered from already existing websites. Back tracking the source of such content will become obsolete and also changes to the source are difficult to be tracked. So to solve this problem document based content management (DCMS) is very much essential. Stored documents in DCMS comprises of huge amount of data so there is need for document clustering. Previous works on clustering documents have no consideration for the semantic information as they consider only the structural information. In this study, a novel semantic and similarity measure based technique is proposed that concurrently considers both structural and semantic information of document. Semantic analysis based clustering is applied to the text documents and then similarity measure is devised among the documents based on machine learning algorithms using Apache hadoop. In order to achieve accurate clustering and efficient retrieval, initially the documents are stored in hadoop distributed file system and they are clustered using K-means algorithm.Then the clustering is also done using concept matching technique and time for formation of clusters were plotted and compared.